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Acronyms 


CM 

Configuration Management 

DTRA 

Defense Threat Reduction Agency 

DUTs 

Devices Under Test 

ESA 

European Space Agency 

FPGA 

Field Programmable Gate Array 

FWHM 

Full Width Half Max 

10 

Input/Output 

LET* 

Linear Energy Transfer Threshold 

NEPP 

NASA Electronic Parts and Packaging 

NSREC 

Nuclear and Space Radiation Effects Conference 

PEMs 

Plastic Encapsulated Microcircuits 

PI 

Principal Investigator 

POF 

Physics of Failure 

RADECS 

Radiation Effects on Components and Systems 

SEE 

Single Event Effects 

SEL 

Single Event Latchup 

SEU 

Single Event Upset 

SOC 

Systems on a Chip 

VHDL 

Very High Speed Integrated Circuit (VHSIC) Hardware 
Description Language 
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Outline 
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- Programmatic constraints 

• Device Considerations 

- A word on data collection 

• Test Set Considerations 

• Facility Considerations 

• Logistics 

• Contingency Planning 

• Test Plan Outline 

• Summary 
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Motivation 

* General guidance on performing SEE testing 
such as the JESD57 [1], ASTM F1 192 [2], and 
ESA [3] documents have been in place for 
decades. 

* However, little guidance exists for developing the 
appropriate SEE test plan. 

* In 201 1 , LaBel et a/, presented thoughts on this 
topic at the RADECS Conference short course [4]. 

- Unfortunately, this is not an archival record and not 
widely available to the U.S. aerospace community. 

- We attempt to rectify that here by providing an overview 
of SEE test planning, with updates from the 201 1 
presentation. 

[1] http://www.jedec.org/standards-documents/docs/jesd-57 

[2] http://www.astm.org/Standards/F1192.htm 

[3] https://escies.org/download/specdraftapppub?id=995 (ESCC Basic Specification No. 25100) 

[4] K. A. LaBel et al., “Single Event Effect (SEE) Test planning 101,” RADECS Conf. Short Course, Seville, Spain, 2011. 
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Abstract 


* While standards and guidelines for performing 
SEE testing have existed for several decades, 
guidance for developing SEE test plans has not 
been as easy to find. 

* In this presentation, the variety of areas that need 
to be considered ranging from resource issues 
(funds, personnel, schedule) to extremely 
technical challenges (particle interaction and 
circuit application), shall be discussed. 

* Note: we consider the approach outlined here as 
a “living” document: 

- Mission-specific constraints and new technology related 
issues always need to be taken into account. 
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Scope and Limitations 

* This is a presentation on SEE Test Plan (or 
Test Planning) development. 

* It is NOT: 

- How to test or testing methodology, nor, 

- A detailed discussion of technology, nor, 

- New material on new effects. 

* It is: 

- An introductory discussion of the items that go into 
planning an SEE test that should complement the 
SEE test methodology used. 

* Material will only cover heavy ion SEE testing 
and not proton, LASER, or other, although 
many of the discussed items may be 
applicable to these other test sources. 
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General 


for a SEE Test Plan 


• Introduction and objectives (or requirements) 

• Detailed device information 

• Documentation 

- Block diagrams, circuit diagrams, cabling diagrams, 
datasheets, etc... 

- Photos of device and test set 

• Equipment list 

- Packing and shipping information 

• Test methodology and data capture 

- Including Data Storage Structure 

• Configuration management 

- Data backup and distribution plan 

• Personnel and logistics 

• Data analysis plan 

• Contingency plan 


Deliverable to NASA Electronic Parts and Packaging (NEPP) Program to be published on nepp.nasa.gov originally presented by Kenneth A. LaBel at the 
2014 Government Microcircuits Applications & Critical Technologies Conference (GOMAC) Tech, Charleston, SC, March 31 to April 4, 2014. 


7 


SEE Test Plan Factors 

* Three factors are weighed in developing the test 
plan: 

- Technical (all the engineering and science), 

- Programmatic (all the cost and schedule), and, 

- Logistics (all the constraints of being off-site and having 
appropriate documentation/configuration). 

* Keeping these considerations in mind, a SEE test 
plan combines 

- The information ranging from requirements to 

- Resources to technical approach to 

- Post-testing analysis. 

* These are the main talking points of this 
presentation. 
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Requirements - 
Dual and Competing Nature(s) 


• Programmatic 

- Cost 

- Schedule 

- Personnel 

- Availability 

- Criticality 

- RISK! 


• Technical 

- Device 

- Packaging 

- Beam/facility 

- Application 

- Data Capture 


Dual Nature 2: Flight Project versus Research 

How we plan and prepare for a test will also vary 

with this trade space. 

All tests are driven by requirements and objectives in 

one manner or another. 
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Flight Project Requirements 

• When planning a test for a flight project, considerations may 
include: 

- Acceptance criteria 

• Error or fail rate (System or Device) 

- System availability may be appropriate, as well 

• Minimum device hardness level 

- LET th , for example 

• Error definition and application information 

- User application(s) 

• Circuit 

- We note that “test as you fly” is often recommended, but not necessarily 
appropriate for an accelerated beam test. 

• Criticality 

- Programmatic constraints. 

• The bottom line is that flight project tests are usually application 
specific and designed to get a specific answer such as: 

- Is the SEL threshold higher than X? or 

- Will I see an effect more than once every 10 days? 

The key concept for developing the SEE test requirements is to 
bound the risk related to mission interruption , anomalies , or failure. 
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Research Requirements 

• These are less specific than requirements for flight projects 
and may include: 

- Generic technology/device hardness or physics of failure, 

- Application range, 

- Angular exploration, 

- Frequency exploration, 

- Beam characteristics such as ion/energy/range effects, 

- Error propagation, charge sharing, etc., 

- Circuit effects, and, 

- Test methodology evaluation. 

• Programmatic constraints still need to be considered (beam 
time, costs, etc.). 

• The bottom line is that all requirements and objectives 
should be “in plan,” i.e., considered prior to test and 
included in test plan development. 
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Resource Estimation 

* Many factors will weigh in to actual resource (re: 
cost and schedule) considerations including: 

- Complexity of device/test and preparation thereof, 

- Facility availability (and time allotment), 

- Urgency of test, and, 

- Funds availability, and so forth. 

* It is important to remember to work with the 
facility of choice to get scheduled in advance - 
last-minute access is rare. 

* Schedules should be developed and included that 
include all phases of testing from requirements 
definition to completed report to the customer. 

* The next chart provides an example of cost 
considerations. 
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Cost Estimation Factors 


• Labor 

- Pl/team lead 

- Test engineers/technicians 

• Electrical, mechanical, VHDL, software, cabling, etc. 

- Test set verification and debugging 

- Test performance (pay attention to overtime needs) 

- Data analysis 

- Report and plan writing 

• Non-recurring engineering costs 

• Board fabrication and population 

• Device thinning/delidding 

• Cables, connectors, parts, miscellaneous 

• Test equipment purchase/rental 

• Facility Costs 

- Note that estimating the amount of beam time required is non-trivial - 
modes of operation, ions, temperature, power, etc. all factor into the 
test matrix and need to be prioritized. 

• Travel 

• Shipping 
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Device Constraints 

* DUTs can range from very simple transistors to 
the most complex SOC. 

- This range implies test set implementations can vary 
just as widely. 

* Starting points are always two-fold : 

- The datasheet for the DUT and 

- The application/suite of applications that will operate in 
space or are of research interest. 

* We note that implementing a test set hinges 
greatly on the DUT type and requirements. 
However, a detailed discussion of this is out of 
scope for this talk. 
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Note on DUT Constraints 


* While the usual “test as you fly” concept is applicable 
to simple devices (for example, an operational 
amplifier application), the more complex the device 
(for example, an FPGA), the more challenging it 
becomes to design an appropriate test (and build a 
reliable tester). 

- This is due to the accelerated nature of the SEE test versus space 
particle rates. Actual flight designs might not be appropriate for 
gathering sufficient information because their state-space and 
functional block coverage may not be statistically significant during 
short beam test run times. 

* It is important to note that due to the huge number of 
application options modern complex devices are 
capable of, most SEU testing is geared to provide 
application-specific information. 

- Interpreting this data is challenging, at best, for other applications. 
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DUT Parameter Space 

• DUT parameter space may include multiple items found on 
datasheets: 

- Electrical performance, 

• Frequency, timing, load, drive, fanout, 10, ... 

- Application capability/operating modes, 

• Processing, configuration, utilization... 

- Power, 

- Environmental characteristics, and so on. 

• Mission-specific testing will limit the parameter space as part of 
the requirements. 

- Research tests must consider the overall application space of the 
DUT and determine priorities for configuration of tests. 

• We note that device sample size is also considered and may be 
limited due to resource or other constraints. 

- Good statistical methods are still recommended. 

- Lot qualification issues should be considered. 

• Key features, device markings, etc. should be included. 
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Predicting what you might observe - 

DUT SEE Categories 


* An analysis of the types of SEE that may occur 
during irradiation is required. 

- This may be called a error/failure mode analysis. 

- Predicted type, and even frequency of SEEs, will drive 
the data capture requirements discussed later as will 
error propagation/visibility. 

* An analysis should include: 

- Upset (single, multiple, transient, functional interrupts, 
etc.) and destructive issues, as well as, 

- Mission-specific objectives (e.g., application 
requirements or destructive test only). 

* Note: Reviewing existing data on similar device 
types and technologies may help this process. 
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Test Set Requirements 

• Test set requirements are a set of derived requirements from the 
mission/DUT/facility requirements. 

- Example: requirement for a test in vacuum may be different than one in 
air. 

• Knowing how a DUT performs is one thing, but defining 
requirements for a test system is clearly separate. 

- Test set requirements should encompass actual application range or 
have sufficient flexibility such that modifications can be made on site 
easily. 

• Mission Requirements generally have ranges of operation. 

- The test set should accommodate this range in areas such as: 

• Min, max, and typical (speed, temperature, voltage), 

• Variety of inputs, 

• Static and dynamic test capabilities, and, 

• Output loading. 

• We note that a test plan should provide full details, 
schematics, figures, photos, etc. of test method/set. 
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Data Requirements 

• Data requirements may be broken into two categories: 

- Data capture, and, 

- Data analysis. 

• Data capture, in this context, is not how you capture the 
data, but the requirements/items that should be considered 
for capture. 

• Data analysis is the other end of the picture: everything 
from the system-wide flow of the data, what format it is 
being captured in, and what are the requirements for 
analyzing this data (real-time and post-testing, as well as 
planning how this should be implemented. 

• We suggest treating radiation data much like a spacecraft 
treats science data: a telemetry and command system. 

- Utilize as many reliable design practices as possible to have 
confidence in the results. 
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Processing the Data 

* Every plan should include a discussion of how 
the data will be processed whether it’s: 

- FWHM for transients, 

- Physical mapping of errors and multiple bit events, or 

- Any of the myriad of data events in between. 

* Requirements need to be divided into three 
categories: 

- Real-time (during the beam run), 

- Off-line (between beam runs), and, 

- Post-processing (after completion of beam runs). 
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Facility Considerations 

* Each test facility has unique requirements 
including: 

- The beam characteristics (energies, profile, etc.) - this 
drives issues such as DUT deprocessing, 

- DUT board mounting (vacuum/air, mounting plate, 
thermal interface, etc.), 

- Cabling (distance, feedthrus, etc.), and so on. 

* Test plans need to take these into consideration 
to ensure reliable testing. 



Acid etch/de-pot 
plastic encapsulated 
microcircuits (PEMs) 


Avoid the dreaded 
CABLE CADAVER 
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Configuration Management 


• The rule here is simple: know and document what 
you have, what you are using, and how you are 
using it. This ranges from cabling all the way to 
coding! 

- CM defines which version you have and making sure 
you bring the tools to modify if needed. 

• Examples: which VHDL code is the final one for either the 
test set or DUT (if applicable)? 

• Each team member is responsible for CM. 

* Data backup is related. 

- Make sure you have a plan for storage of multiple copies 
of the data, who is responsible, and what happens for 
post-processing. 
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Logistics 


• While non-technical, logistics related to test 

planning and writing a test plan are no less 

important. 

* Areas for consideration in no particular order: 

- Test team member contact info (cell phones, hotels, 
flights, etc.), 

- Facility contact information including maps for 
“newbies,” 

- Contact information for key people at home site, 

- Equipment list including spares, 

* Don’t forget datasheets! 

- Shipping/transport of equipment (cost, tracking, etc.), 
and, 

- Roles and responsibilities of the team. 
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Contingency 


* Contingency is required for several reasons: 

- Test set does not work; 

- Test set does not work as well as expected; 

- Error signatures are different than anticipated; or, 

- Facility may have an “issue” such as the beam going 
down. 

* A good plan will include: 

- Prioritization of tests planned (which devices, which 
tests); 

- Limits on debug time to make a decision to test, move to 
a later test timeslot, etc.; and, 

* Example: if after 1 .5 hours no significant progress is made, 
go to backup device. 

- Backup devices (in case test ends early or other 
device/test doesn’t work properly). 
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SEE Test Plan Outline - Summary 

• Introduction and objectives 

• Detailed device information 

• Documentation 

- Block diagrams, circuit diagrams, cabling diagrams, 
datasheets, etc. 

- Photos of device and test set 

• Equipment list 

- Packing and shipping information (detailed) 

• Test methodology and data capture 

- Including data storage structure 

• Configuration management 

- Data backup and distribution plan 

• Personnel and logistics 

• Data analysis plan 

• Contingency plan 
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Summary 



* This presentation was designed to provide the 
user the basic thought processes required to 
develop a successful test plan focusing on: 

- Technical issues, 

- Logistics issues, and, 

- Programmatic issues. 

* We note that every DUT is different and no two 
test plans look alike. 


Deliverable to NASA Electronic Parts and Packaging (NEPP) Program to be published on nepp.nasa.gov originally presented by Kenneth A. LaBel at the 
2014 Government Microcircuits Applications & Critical Technologies Conference (GOMAC) Tech, Charleston, SC, March 31 to April 4, 2014. 


26 


Acknowledgements 

* The authors would like to acknowledge our 
sponsors and collaborative agencies: 

- NASA Electronic Parts and Packaging (NEPP) Program, 
and, 

- The Defense Threat Reduction Agency (DTRA). 

* We would also like to thank: 

- Martha O’Bryan for her presentation support, and, 

- Lew Cohn for his valuable discussions. 



Deliverable to NASA Electronic Parts and Packaging (NEPP) Program to be published on nepp.nasa.gov originally presented by Kenneth A. LaBel at the 
2014 Government Microcircuits Applications & Critical Technologies Conference (GOMAC) Tech, Charleston, SC, March 31 to April 4, 2014. 


27 


